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What the Harvard/Mathematica 
Study Says About Vouchers and Low- 
Income African-American Students 

Kirk A. Johnson, Ph.D., and Krista Kafer 



In February 2002, researchers at Harvard 
University, Mathematica Policy Research, Inc. 
(MPR), and the University of Wisconsin 
released the results of a three-year study to 
determine the effects of voucher-like scholar- 
ships on low-income student achievement in 
New York City. 1 In just three years, the vouch- 
ers offered by the School Choice Scholarships 
Foundation (SCSF) were found to have had 
impressive effects, especially for African- 
American low-income students. Specifically: 

• Standardized reading and math test scores 
for black students who had used the 
vouchers (worth up to $1,400 each year) 
to attend private schools for three years 
were 9.2 percentile points higher than 
those of comparable black students who 
did not attend a private school. 

• Overall test scores for black voucher recip- 
ients who attended a private school for at 
least one of the three years were, on aver- 
age, 7.6 percentile points higher than those 
of black students who had never attended 
a private school. 

• Parental satisfaction with their child’s 
school was higher among parents of stu- 
dents who attended a school of choice. 
When asked to assign a grade to their chil- 



dren’s school, 42 percent of voucher par- 
ents gave their school an “A,” while only 10 
percent of the parents of the control group 
public school students did likewise. 

Thus, voucher-like scholarships have signifi- 
cant effects on the education of low-income 
students. Even black students who had 
attended private school for only one or two 
years did better than their public school peers 
did on standardized reading and math tests. 
This CDA Report will first discuss how the 
School Choice Scholarships Foundation pro- 
gram selected students to receive the vouchers 
and then will review the results of the study 
published by Mathematica. 

BACKGROUND 

In 1996, the School Choice Scholarships 
Foundation, a philanthropic, privately funded 
voucher program, began collecting money to 
fund scholarships for students in New York 
City public schools to attend a private or paro- 
chial school of choice. Since 1997, 1,300 
vouchers have been awarded to the children of 
low-income families; these vouchers, which 
have a maximum value of $1,400 annually, are 
redeemable for at least three years. To be eligi- 
ble for the vouchers, the student: 



1. Daniel P. Mayer, Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell, “School Choice 
in New York City After Three Years: An Evaluation of the School Choice Scholarships Program Final Report,” 
Mathematica Policy Research, Inc., Report No. 8404-045, February 19, 2002. 
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Black Voucher Students Score Higher Than 
Non-Voucher Students in Reading and Math 



Control Student 



At Least One-Year 
Voucher Student 



Control Student 



Three-Year 
Voucher Student 




Iowa Test of Basic Skills National Percentile Flanks, Composite Reading and Math 



Source: Daniel P. Mayer, Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell, ‘’School Choice 
in New York City After Three Years: An Evaluation of the School Choice Scholarships Program Final Report,” 
Mathematica Policy Research, Inc., Report No. 8404-045, February 19, 2002. 



• Must be entering the first through fifth grades; 

• Must be attending a New York City public 
school; and 

• Must qualify for the federal school lunch pro- 
gram (be from a low-income family). 2 

Between February and April 1997 alone, SCSF 
received some 20,000 applications for the vouch- 
ers. The overwhelming response necessitated that 
the foundation employ objective criteria for 
awarding the scholarships. Once the criteria had 
been established, a series of random lotteries was 
conducted in May 1997 to select the students who 
would receive the vouchers. 



During the application and eligibility verifica- 
tion process in 1997, students were asked to take a 
baseline achievement test — the Iowa Test of Basic 
Skills (ITBS) — in reading and math. (Kindergart- 
ners applying for first grade scholarships were 
exempt.) Their parents filled out a series of ques- 
tionnaires about their satisfaction with their chil- 
dren’s current school, their involvement with their 
childrens schooling, and demographic characteris- 
tics. Researchers at Harvard University, Mathemat- 
ica Policy Research, and the University of 
Wisconsin used this information to assess the 
effectiveness of vouchers in raising student 
achievement. 



2. The income eligibility levels for the U.S. Department of Agriculture’s free lunch program and reduced price lunch program 
in 2001-2002 for a family of four are, respectively, $22,945 (130 percent of the federal poverty line) and $32,653 (185 
percent of the poverty line). See U.S. Department of Agriculture, Food and Nutrition Service, “School Programs: Income 
Eligibility Guidelines,” at http://www.jns.usda.gov/cnd/Lunch/Govemance/Notices/01-02iegs.htm. 
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Most of the students who applied (about 70 
percent) scored below the citywide median score 
on the ITBS; thus, most of the recipients of the 
vouchers would be students who had scored 
below average. The SCSF, in consultation with the 
evaluators for the Mathematica study, decided that 
85 percent of all the vouchers awarded would go 
to below-average students. Other than this restric- 
tion, the chance of an individual students winning 
a scholarship was the same for all the students 
who were eligible. 

The basic design of the Mathematica study gave 
the researchers a unique opportunity to assess the 
effectiveness of a working voucher program. 3 The 
randomized lottery effectively assigned the stu- 
dents into two groups: the “experiment/treatment” 
group who received the vouchers and the “control 
group” who did not. This design is very similar to 
the way researchers in the medical community 
assess the effectiveness of new procedures or new 
drugs; it is generally considered to be the “gold 
standard” of research protocols. 4 

WHAT THE RESEARCH SHOWS 

The experiment/treatment group chosen by the 
lottery was comprised of 1,000 families in New 
York City, while the control group had 960 fami- 
lies. The random nature of the lottery assured that 
the families in each group would not be statisti- 
cally different from the others in terms of family 
background, income, or student achievement. 
Since all the families, as part of the application 
process, had to fill out background survey forms 
and take the ITBS tests, comprehensive baseline 
information was readily available to the research- 
ers for purposes of comparison. 

In the spring of 2000, both the experiment and 
control groups were invited back to complete 
third-year follow-up questionnaires and tests. 



These tests were conducted much as the baseline 
survey was conducted. 

ITBS scores are reported using National Percen- 
tile Rankings (NPR) instead of raw scores. The per- 
centile rankings show how well students perform 
in a particular subject relative to everyone else in 
the country who took the test that year. If a stu- 
dent received an NPR of 50 on the ITBS reading 
exam, for example, he or she had scored higher 
than 50 percent of all students taking that test. 

The Mathematica researchers reported each stu- 
dent’s reading and math results separately, as well 
as a composite test score that averaged the scores 
on both tests. 

The 2000 ITBS test scores showed that the 
vouchers had had significant effects, which were 
most pronounced for black students. 5 Chart 1 
shows the average difference in the composite 
ITBS test scores. Black students who used the 
voucher to go to a school of choice for three years 
had a composite NPR of 26.83 on their follow-up 
tests, compared with a control group of black stu- 
dents who had an NPR composite score of 
17.60 — a 9.23 percentile point difference. The 
scores of all black voucher students who had 
attended a school of choice for at least one year 
were significantly higher than those of the control 
group. These “part-time” voucher students had a 
composite NPR of 25.37, compared with a com- 
posite score of 17.82 for the control group — a dif- 
ference of 7.55 percentile points. 6 

The results are even more pronounced for math 
achievement, as shown in Chart 2 and Table 1. 
Black voucher students who stayed in a school of 
choice for three years scored a full 1 1.80 percentile 
points higher on the math test than did the control 
group, and voucher students who had attended a 
private or parochial school for at least one year 
scored 9.65 percentile points higher in math. 7 



3. These researchers have analyzed similar experiments elsewhere, including Washington, D.C., and Dayton, Ohio. Their 
findings will be offered in William G. Howell and Paul E. Peterson, with Patrick J. Wolf and David E. Campbell, The Edu- 
cation Gap: Vouchers and Urban Schools (Washington, D.C.: Brookings Institution Press, forthcoming 2002). 

4. For a fuller description of experimental research in the social sciences, see Catherine Hakim, Research Design: Strategies and 
Choices in the Design of Social Research , Vol. 13, Contemporary Social Research Series (New York: Routledge, 1986), pp.- 
101-116. 

5. Latino students receiving the vouchers exhibited statistically similar scores after three years. Other racial or ethnic groups 
in the experiment did not have enough students participating to conduct a statistically sound comparison. 

6. See Mayer et a/., “School Choice in New York City After Three Years,” Table 20, p. 63. 

7. Ibid., Table 22, p. 65. 
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Black Voucher Students Have Higher Achievement in 
Reading and Much Higher Achievement in Math 



Reading 



Math 
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Iowa Test of Basic Skills National Percentile Ranks 

Source: Daniel P. Mayer, Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell, "School Choice in New York 
City After Three Years: An Evaluation of the School Choice Scholarships Program Final Report” Mathematica Policy Research, Inc., 
Report No. 8404-045, February 19, 2002. 




The differences are somewhat lower, but still 
statistically significant, for reading achievement. 
Three-year voucher students scored 6.66 percen- 
tile points higher than did the control group, and 
students who had used the voucher for one or two 
years scored 5.45 percentile points higher. 8 

AN OVERVIEW OF OTHER 
CHOICE ACTIVITY 

The findings of this Harvard/Mathematica study 
on the School Choice Scholarship Foundations 
working private voucher program in New York 
City add to a growing body of research on the 
effects of school choice on student achievement. 
Since the SCSF was founded some six years ago, 
similar funds have offered voucher-like scholar- 



ships to students in Washington, D.C., and Day- 
ton, Ohio. The initial studies of these programs 
show promising results as well, 9 as does the 
research on the students using a publicly funded 
voucher program in Cleveland, Ohio. 10 

As the effects of school choice — ranging from 
greater satisfaction to higher achievement — are 
better documented and gain public recognition, 
the demand for choice programs grows. As of June 
2001, charter school laws had been enacted by 37 
states and the District of Columbia; 10 states had 
initiated publicly sponsored private school choice 
programs, from vouchers to tax credits; and more 
than 50,000 students had been able to attend a 
public, private, or parochial school of choice due 



8. Ibid., Table 21, p. 64. 

9. See, for example, William G. Howell, Paul E. Peterson, Patrick J. Wolf, and David E. Campbell, “School Vouchers and Aca- 
demic Performance: Results from Three Randomized Field Trials,” Journal of Policy Analysis and Management, Vol. 21, No. 2 
(2002), pp. 207-233. See also Howell and Peterson, The Education Gap. 

10. Jay P. Greene, William G. Howell, and Paul E. Peterson, “Lessons from the Cleveland Scholarship Program,” in Paul E. 
Peterson and Bryan C. Hassel, eds., Leammgfrom School Choice (Washington, D.C.: Brookings Institution Press, 1998). 
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Black Voucher Students Have Higher Achievement 
Than Black Control Students 
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to the efforts of the almost 100 privately funded 
scholarship programs across the country. 11 

In February 2002, hundreds of choice support- 
ers, primarily parents and children, lined up in 
front of the U.S. Supreme Court to show their sup- 
port for the publicly funded Cleveland Scholarship 
and Tutoring Program. Inside, the justices heard 
oral arguments regarding Zelman v. Simmons-Har- 
ris, a federal appeals court ruling that struck down 
the Cleveland voucher program as unconstitu- 
tional. The Supreme Court’s ruling on the Cleve- 
land program’s constitutionality is not expected 
until late June. If the program is upheld, choice 
programs are likely to grow in number as viable 
options for low-income parents who want their 
children to excel in school. 

In 2001, President Bush signed a tax relief bill 
that expands opportunities for parents to build 
tax-free savings for their childrens education 
expenses in education savings accounts (ESAs). 
This year, he has proposed a refundable tax credit 
that would enable children trapped in failing 
schools to attend schools of choice. He also has 
proposed a $50 million Choice Demonstration 
Fund that would support research projects to 
develop, implement, and evaluate innovative 
school choice programs for low-income students 
that would enable them to attend private, magnet, 
charter, and other public schools. 



In addition, the No Child Left 
Behind Act, which was signed into 
law on January 8, 2002, mandates 
public intra-district school choice for 
children in persistently failing 
schools. It also gives poor children 
the right to choose a supplementary 
service provider, either public or pri- 
vate, and guarantees that parents are 
given information on the effective- 
ness of their schools. 13 This informa- 
tion will feed the demand for better 
schools, but all parents will need to 
be empowered to act upon the infor- 
mation they receive to choose 
schools of excellence for their chil- 
dren. 

CONCLUSION 

The positive findings of the recently released 
study by researchers at Harvard University, Mathe- 
matica Policy Research, and the University of Wis- 
consin on how vouchers have affected 
achievement among poor students in New York 
City should come as no surprise. It is an example 
of a growing body of research that reveals both the 
achievement gains voucher-style approaches offer 
and the greater satisfaction among parents who 
use those vouchers to send their children to a 
school of choice. 

In just three years, the effects of the School 
Choice Scholarships Foundation vouchers were 
sizeable. African-American students who prima- 
rily were poor and scored below average on the 
Iowa Test of Basic Skills and who used the SCSF 
vouchers to attend a private or parochial school 
improved their test scores in math by almost 10 
percentile points in just one year. That type of 
improvement is significant, both for the students 
and their parents, and for the nation. 

— Kirk A. Johnson , Ph.D., is a Senior Policy Analyst in 
the Center for Data Analysis, and Krista Kafer is 
Senior Policy Analyst for Education, at The Heritage 
Foundation. 



11. See Center for Education Reform, “Charter School Highlights and Statistics,” at http://www.edreform.com. 

12. RL. 107-110. 

13. For more information, see Krista Kafer, “A Small But Costly Step Toward Reform: The Conference Education Bill,” Heritage 
Foundation Web Memo, December 13, 2001, at http://www.heritage.org/shorts/20011213education.html. 
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